A novel hack involving Large Language Models (LLMs) has emerged, exploiting adversarial suffixes to deceive models into generating perilous responses. Such jailbreaks can trick LLMs into providing intricate instructions to a malicious user for creating explosives, orchestrating a bank heist, or facilitating the creation of offensive content. By evaluating the perplexity of queries with adversarial suffixes using an open-source LLM (GPT-2), we found that they have exceedingly high perplexity values. As we explored a broad range of regular (non-adversarial) prompt varieties, we concluded that false positives are a significant challenge for plain perplexity filtering. A Light-GBM trained on perplexity and token length resolved the false positives and correctly detected most adversarial attacks in the test set. LLMs like ChatGPT, BARD, LLaMA-2-Chat, Claude, and other such models have caused rapid responses to complex queries in natural language to become easily accessible. Safeguards to reduce model abuse have helped align them with ethical standards (Ouyang et al., 2022). For instance, models can reply with a refusal when illicit queries are made (OpenAI, 2023). Attempts to circumvent such alignment safeguards have emerged in the form of adversarial prompt engineering and LLM jailbreaks (Wei et al., 2023).